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ABSTRACT 

Careers guidance in Scotland is about to undergo major 
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a national, ' all-age careers guidance service. Among its aims. Careers 
Scotland seeks to establish national quality standards for careers provision. 
This briefing draws on a recently completed study of the ef f ectiveness of 
careers services in Scotland to consider some of the issues involved in 
assessing the quality and effectiveness of careers guidance work. (Author) 
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How Would You Know? Assessing the 
Effectiveness of Careers Guidance 
Services 

by Cathy Howieson and Sheila Semple No. 22, October 2001 




Careers guidance in Scotland is about to undergo major structural 
change with the creation of “Careers Scotland” which will provide a 
national, all-age careers guidance service. Among its aims, Careers 
Scotland seeks to establish national quality standards for careers 
provision. This Briefing draws on a recently completed study of the 
Effectiveness of Careers Services in Scotland to consider some of the 
issues involved in assessing the quality and effectiveness of careers 
guidance work. 



► Assessing the effectiveness of careers guidance is a difficult task. A number of 
approaches are possible, each has its strengths but none provide a comprehensive 
assessment. 

► Quantitative monitoring of work activities and client feedback were the most 
common methods used in Careers Service Companies (CSCs) in Scotland. Each 
approach has limitations in what it can assess. 

► Other methods such as internal review of careers advisers’ professional practice 
by peers or managers and learning outcome measures were limited in the CSCs. 

► CSCs had become more accountable for their activities through the monitoring of 
business activity targets and progress had been made in quality assurance. But 
these approaches did not adequately assess the quality of inputs and how well 
they met identified client needs. 

► The research developed two models as examples of a comprehensive and 
integrated approach to assessing all aspects of careers service work, 
incorporating a range of assessment methods as appropriate. 
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Introduction 

Careers guidance in Scotland is about to undergo major 
structural change with the creation of “Careers Scotland” 
- a national organisation with the remit to ensure the 
provision of an all-age guidance service. Careers Scotland 
has been set up following the Duffner Review of the 
Careers Service and subsequent consultation by the 
Scottish Executive (Scottish Executive, 2000; 2001a; 
2001b). 

The Duffner Report noted the considerable 
commitment among Careers Service Companies (CSCs) to 
the measurement of the quality and effectiveness of their 
work but recommended the need to establish national 
minimum standards, to update performance measures and 
to define professional standards for careers advisers - 
recommendations accepted by the Scottish Executive and 
endorsed in the public consultation. Currently, as part of 
the move to Careers Scotland, a “Quality and Standards” 
Transition Team has been set up. 

Against this background, this Briefing discusses how 
the effectiveness of the work of the Careers Service (CS) 
might be assessed and the sorts of issues that the “Quality 
and Standards” Transition Team need to take into account 
in developing national standards for Careers Scotland. 

Approaches to measuring the performance 
of Careers Scotland 

Measuring the effectiveness of careers guidance and the 
work of careers guidance services is a difficult task; some 
respondents in the research questioned whether it is 
feasible. There are issues, for example, about what is to be 
measured, what constitutes success, whether it is possible 
to establish cause and effect and how to take account of 
other intervening factors such as careers education inputs, 
and family and social influences. A number of approaches 
can be identified, each of which is better suited to 
measuring certain aspects of Careers Service work than 
others: quantitative monitoring of activities; client and 
stakeholder feedback (or customer satisfaction measures); 
learning outcomes measures; and longitudinal measures of 
impact (see eg Killeen and Kidd, 1992; Killeen, 1996). 

The research found that the quantitative monitoring of 
work activities to meet Scottish Enterprise requirements 
and activities linked to the Scottish Quality Monitoring 
System (SQMS) and Investors in People (IIP) formed the 
bulk of performance measurement carried out in Careers 
Service Companies. The SQMS award is held by all 
Careers Service Companies in Scotland. It uses 
quantitative measures, client and stakeholder feedback 
and both internal and external review to measure 
performance. 

Quantitative monitoring of activities 

®^.ince 1996, Careers Service Companies have been 
squired by the Scottish Executive to produce targets and 
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business activity measures ie quantitative monitoring of 
activities. All CSCs therefore collected Business Activity 
statistics on the incidence of certain specified activities 
(eg the number of interviews carried out in school and 
with the unemployed and with young workers, the number 
of young people placed etc). Such quantitative monitoring 
of work activities is one of the simplest approaches and 
can answer questions such as “how many, how often, 
when, who?” It can describe what happens but, critically, 
it cannot indicate how well it happens. This is the key 
limitation of the Business Activity statistics produced by 
CSCs. 

Related to the Business Activity statistics are 
Efficiency Indicators which identify unit costs for specific 
activities. However, their usefulness depends on accurate 
and uniform recording across CSCs but the research found 
that recording did not always meet this criteria. 

Careers Service Companies are also obliged to 
produce the School Leaver Destination Return (SLDR) for 
the Scottish Executive. The SLDR alone is a blunt 
measure of effectiveness of Careers Service work. Levels 
of unemployment among school leavers, for example, may 
say more about the local labour market and social issues 
than about Careers Service provision. 

Client and stakeholder feedback (or customer 
satisfaction measures) 

Customer/client feedback had been in increasing use in 
Careers Service Companies prior to SQMS but the 
requirements of the SQMS audit had made it compulsory 
for all Careers Service Companies. The most common 
approach was the use of questionnaires although several 
Careers Service Companies had used focus groups. 

Customer/client feedback offers a measure of clients’ 
perceptions of the appropriateness and value of the service 
to them but it has certain limitations. Clients’ expectations 
of careers guidance services will have an impact on their 
response as will their awareness of their own careers 
guidance needs. It is important to recognise that clients’ 
demand for certain guidance services is not necessarily 
identical to their guidance needs. The mechanics of how 
client opinion is gathered are critical, including the timing 
of the event and the questions asked of clients. The 
Scottish Executive has provided some central support in 
training and templates for feedback, but the research still 
found a high degree of variation in how this was done, 
and in the appropriateness of the instruments used. Also, 
it is often very difficult to obtain the views of non-users of 
the Careers Service. This was less commonly done by 
CSCs although the views of non-users can be particularly 
useful. 

Learning outcomes measures 

Learning outcomes measures take a more short-term 
approach to assessing the effectiveness of Careers Service 
inputs. They seek to ask what changes have happened in 
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the client's knowledge, skills, attitudes and behaviour as a 
result of Careers Service work. They may include “before 
and after” measures of attitudes and perceptions linked to 
specific Careers Service interventions. 

Learning Outcomes measures were used relatively 
rarely in the Careers Service. Where they were, usually 
“home-grown” quizzes or questionnaires were used rather 
than commercially produced instruments. Only 
occasionally was a “before” baseline measure gathered as 
part of this. 

Internal professional review 

All CSCs monitored the activity targets set for individual 
staff and/or areas; this might be done on a weekly, 
monthly or bi-monthly basis depending on the CSC 
concerned. Targets meant staff felt more accountable for 
their work but they recognised that they monitored 
volume rather than quality. Internal review of professional 
practice was limited: review of action plans was the most 
common method used but review of client records was 
less common and observation of interviews was least 
developed. There was a great deal of uncertainty about 
how to develop this and the research found that there was 
no single common approach in use to assess professional 
practice. The study identified concern among managers 
about the impact of internal review on staff and CSCs 
were generally tentative about this area. 

There are issues about the subjectivity of internal 
review and it can also be difficult for staff who are acting 
as reviewers to be critical (however constructively) of 
colleagues. But professional review allows evidence to be 
understood in its context and with an understanding of the 
professional issues involved. 

External review 

External review of Careers Service activities by 
government bodies, consultants or researchers is another 
approach to reviewing effectiveness. The value of external 
reviews depends on their remit, their objectivity and the 
knowledge and skills of the individuals or organisations 
carrying them out. 

A number of Careers Service Company Boards had 
commissioned external consultants or researchers to 
review performance and make recommendations. The 
impact of such external reviews appeared to be varied, 
with some examples where recommendations had not 
been implemented and others where Careers Service 
Companies concerned were required to make major 
changes in delivery. 

Longitudinal measures of impact 

Some measures of the economic and social outcomes of 
the work of the Careers Service can only be taken in the 
O ng term. There are considerable methodological 
fficulties with longitudinal studies (Killeen et al , 1992) 



and, from the point of view of CSCs, they are too 
expensive for individual companies to carry out. The 
School Leaver Destination Return only provides evidence 
of the first known post-school destination information. It 
does not take account of transitions, even within the first 
year of leaving school. 

The need to go beyond current practices 

Business Activity targets have made CSCs more 
accountable for their activities and SQMS and IIP had 
made significant contributions to quality assurance in 
Careers Service Companies. But the research indicated 
that there are certain aspects of Careers Service 
Companies’ work that are not measured by these 
approaches, for example, the quality of guidance in an 
interview or the extent to which a particular input meets 
identified needs. 

The variation in provision across and within CSCs 
identified in the study suggests that guaranteed minimum 
standards are not in place for services to clients. SQMS 
focuses mainly on whether effective systems are in place 
rather than assessing how effective provision has been in 
meeting client needs; it also focuses on client demands 
rather than needs. The research suggests that any quality 
management system for careers guidance services requires 
to have a systematic strategy for identifying needs built 
into it. This applies to SQMS and to other models such as 
the European Foundation for Quality Management 
Excellence Model. 

General approaches 

As part of the research, two models to evaluate the 
effectiveness of the CS performance were developed as 
examples of a comprehensive and integrated approach to 
assessing the various aspects of careers service work. 
They incorporate the range of methods outlined above. 

One approach is based on the model of self-evaluation 
used in Scottish schools “How Good is our School at..?” 
(SOEID, 1996). This model enables various aspects of the 
CS work to be covered , for example, “How Good is our 
Careers Service at Careers Interviews?”, “How Good is 
our Careers Service at Working with Employers? 

The second approach is a set of Effectiveness 
Descriptors covering a number of areas of careers work: 
effective careers work with individual clients; effective 
outcomes for individual clients; effective work with other 
clients and organisations; effective work with the 
opportunity structure; effective strategic development; 
and support for effective careers guidance processes. The 
principle is that CSCs audit their careers guidance 
performance against these descriptors of an effective 
Careers Service, set targets that would move them further 
towards them and then measure themselves (and are 
measured) by a variety of methods that assess the extent 
^ to which they had met these targets. 



CES Briefing 

Issues J ° 

► Substantial progress has been made in the quantitative 
monitoring of CS work and in the running of CSCs as 
organisations. Less progress has been made in 
measuring the quality of the careers guidance given to 
clients; this is an area for Careers Scotland to develop. 

► Each of the methods that might be used to assess the 
work of careers guidance services has certain 
limitations. A variety of appropriate methods needs to 
be used. 

► Internal professional review needs to be developed 
and a systematic approach to measuring the standard 
of work of staff and the quality of careers guidance 
interventions is essential. Internal professional review 
needs to become an accepted part of the work of 
Careers Service managers 

► A common concern is the amount of time and 
resources that performance measurement can take. 
But some of the activities should form part of 
everyday professional practice and management 
activities and in designing evaluation systems, care 
should be taken to incorporate existing activities into 
the measurement system. 

► The proper identification of client needs is 
fundamental to the design of effective provision and 
the achievement of satisfactory outcomes for clients. 
The research found that although all CSCs showed 
some evidence of identification of need, there was 
little evidence of a systematic and comprehensive 
approach. If client needs have not been properly 
identified then the most sophisticated performance 
measurement system will be of little value. 

► Areas for collaboration could be explored and Careers 
Scotland will be in a better position than individual 
CSCs to do so through, for example, collaborating 
with the research community to develop a broader 
range of strategies to measure learning outcomes. 

► Few practitioners or managers in Careers Service 
Companies were aware of research evidence on 
effectiveness in careers guidance or made use of it in 
practice. Careers Scotland might consider how 
relevant research can be used to inform professional 
practice and the measurement of performance. 

► Both short and long term assessment of careers 
guidance work is necessary. Longitudinal measures of 
impact are needed. Some data can be collected in 
conjunction with the CS but this is an area where 
centrally funded activity is necessary, for example, to 
use the potential offered by existing longitudinal 
surveys such as the Scottish School Leavers’ Survey 
(SSLS) and the Scottish Household Survey. 
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For further details contact Cathy Howieson, Centre for 
Educational Sociology, tel 0131 651 6241, email 

c.Howieson@ed.ac.uk or Sheila Semple, University of 
Strathclyde, email sheila.m.semple@strath.ac.uk, tel 0141 
950 3326. The views expressed are those of the authors. 



About this study 

The research was commissioned by the Scottish Office in 1997 
to map Careers Service provision across Scotland and consider 
its effectiveness after a period of major change to its 
organisation and management. The research involved: review 
of national documentation and statistics and interviews with key 
informants; collection of documents and statistics and 
interviews with Heads of Service and Careers Advisers in all 17 
CSCs. From this data, a typology of CSCs was created and 
used to select five CSCs for further in-depth study. In each of 
the five CSCs a programme of interviews was carried out with a 
range of CSC personnel; school staff; and representatives of 
Careers Service Boards, LECs, LAs, and Education Business 
partnerships. Three surveys were conducted: S4, S5 and S6 
pupils in 10 schools were surveyed twice, a year apart; a 
second survey covered 300 employers; and the third survey 
was of 60 Training Organisations. 
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